Universal Access To All Knowledge
Home Donate | Store | Blog | FAQ | Jobs | Volunteer Positions | Contact | Bios | Forums | Projects | Terms, Privacy, & Copyright
Search: Advanced Search
Anonymous User (login or join us)
Upload

Reply to this post | See parent post | Go Back
View Post [edit]

Poster: Administrator, Curator, or StaffBrak Date: Aug 31, 2006 4:17pm
Forum: forums Subject: Re: Is there a '24-hour rule' at Archive.org?


Hi,

So this is a cool problem. We do not have any 24 hour ftpd timeouts configured and the master ftpd has been up for weeks on most boxes.

I was going to test this internally, but soon realized that running at 100Mbps for 24 hours meant uploading a 1TB file! Hehe... so I'll test this from home.

Couple of other random theories:
1. ISP's kill a connection after 24 hours of being pegged automatically. This seems unlikely since you mentioned that it's happening to different folks on different ISP's.

2. ISP changes your IP address, causing the connection to go away.

3. Are you using a firewall/cheeseball router thingy. Those have been known to do really stupid things... like time out ssh sessions after 5 minutes. Maybe they do something silly and reset all tcp sessions every 24 hours.

4. OUr own firewall is doing silly things like #3.

I think #3 and #4 are the most likely causes for badness. Home firewalls are notoriously crappy (replacing their firmware with Linux solves most of their problems). They don't have much ram, and dropping connections for the conntrack table is a cheap, if lame, way of not getting support phone calls.

If your box is secure, try uploading while outside of the Linksys.

More in at least 24 hours, I'll start a few uploads tonight and over the weekend.... now if only I had worthy content!

-Joerg

Reply to this post
Reply [edit]

Poster: AlecWest Date: Aug 31, 2006 11:10pm
Forum: forums Subject: Re: Is there a '24-hour rule' at Archive.org?

BRAK WROTE - "I was going to test this internally, but soon realized that running at 100Mbps for 24 hours meant uploading a 1TB file! Hehe... so I'll test this from home."

This may a related issue. My FTP uploads to the archive are throttled back to 42-45k/s - which is why it takes roughly 28-29 hours to upload a 4.3 gig file. My normal upload speeds to other servers are about 10 times that - and one German server I connect to often allows me roughly 175-180k/s uploads.

During a spate of recent upload attempts, I called my ISP (Comcast) to ask if they were throttling back my speed. The tech support rep said no - that it was likely on the server end. Of course, he might have been lying. It could be that "Archive.org" (known for uploads of substantial filesizes) might be on Comcast's s**tlist of connections to throttle.

FWIW, others have also noticed this apparent throttling. But (grin), ISPs could share a common s**tlist.

BRAK WROTE - "1. ISP's kill a connection after 24 hours of being pegged automatically. This seems unlikely since you mentioned that it's happening to different folks on different ISP's."

I'm on Comcast in Portland, Oregon. The other two I communicated with are an Adelphia user in PA and a Verizon user in MD. But again, this could be part of a s**tlist scenario used by more than one ISP. I have a friend in Illinois with his own T3 line and has no throttling issue. And at his speeds, 24-hour timeouts are never reached.

BRAK WROTE - "2. ISP changes your IP address, causing the connection to go away."

My IP is static.

BRAK WROTE - "3. Are you using a firewall/cheeseball router thingy. Those have been known to do really stupid things... like time out ssh sessions after 5 minutes. Maybe they do something silly and reset all tcp sessions every 24 hours."

I never considered that possibility. I use ZoneAlarm but am unaware of any timeout issues with the software.

BRAK WROTE - "4. OUr own firewall is doing silly things like #3."

Could that be the culprit, then?

BRAK WROTE - "If your box is secure, try uploading while outside of the Linksys."

Hmmm ... I use a D-Link router. Next time I upload, I'll disconnect it and run straight off my cable connection to see what happens.

BRAK WROTE - "More in at least 24 hours, I'll start a few uploads tonight and over the weekend.... now if only I had worthy content!"

Thanks ... and I know what you mean. I was certain my last two uploads were public domain ... but one showed up on a superlist and the other showed up on LOC "only" under an abbreviated version of its title (sigh).

I've been tempted to upload the 58-minute "Superman and the Mole-Men." This was the pilot movie to the "Adventures of Superman" TV series in the 50s starring George Reeves. However, unlike the series episodes which were copyrighted by an existing company, the pilot was copyrighted by "Lippert Productions" which is no longer in business. I'm wondering if maybe this might slip through the copyright crack.

Similarly, there were 3 "rocket man" serials from Republic Pictures. Two of them show up on the superlist. But one of them, currently online at the archive (RADAR MEN FROM THE MOON), does not show up on a superlist or LOC ... at least, not under that title. Republic changed the 12-chapter serial into a 12-episode TV series titled COMMANDO CODY - SKY MARSHALL OF THE UNIVERSE - which "is" listed at LOC as being copyrighted. Obviously, RADAR MEN must have slipped through the crack into the public domain by virtue of name change. Likewise, SUPERMAN AND THE MOLE MEN was split into two half-hour episodes in season one of "Adventures" - and retitled "The Unknown People," parts 1 & 2. Could the same "crack" apply to MOLE MEN as well???

But, it's a real bummer to upload something and have it come down. I'm becoming real experienced at that (grin) since I also uploaded the 2 other rocket man serials last year - only to see them come down.

Reply to this post
Reply [edit]

Poster: Administrator, Curator, or Stafftracey pooh Date: Sep 19, 2006 8:55am
Forum: forums Subject: Re: Is there a '24-hour rule' at Archive.org?

did your investigations come up with anything?
any successes?

BTW, which server were you uploading to?
something like "XXX-uploads.archive.org"
or something like "iaXXXXXX.us.archive.org"?

--tracey, webmaster

Reply to this post
Reply [edit]

Poster: AlecWest Date: Sep 23, 2006 7:03am
Forum: forums Subject: Re: Is there a '24-hour rule' at Archive.org?

I'm unsure what "investigations" you're referring to. Suffice it to say that I, and others, have uploaded files to the archives at "iaXXXXXX-type" servers and encountered the problem. Exactly (and I mean "exactly") at the 24 hour mark of uploads, the server cuts the connection. And, upon reconnection, resume is not allowed. And, we have different ISPs and even use different FTP utilities. So, the problem isn't ISP related or even FTP related.

The only way "around" this cut-off is to upload a portion of a large file, manually interrupt the upload prior to hitting the 24-hour "brick wall," then reconnect. That "trick" apparently restarts the cutoff clock and allows for the entire file upload to resume and complete.

Maybe...

At this very moment, I'm attempting to upload an MPEG2 file that, under normal constraints. could be uploaded in less than 15 hours. The server I'm attempting to upload it to is "ia331310.us.archive.org." The problem? It allows me to upload for a few seconds, then slows down, then freezes. When I restart my computer (no kidding) and restart SmartFTP, the process repeats - uploading for a few seconds, then slowing down, then freezing up altogether. And BTW, it isn't my ISP. My browser continues to function just fine (like it is now as I post this message). But my FTP connection to that server totally bites.

There are times when I wonder how many people sincerely want to donate uploaded files to the Archive ... are confronted with maddening problems like this ... and just say, "To heck with it." And I wonder how many files the Archive is deprived of because of this.

Question (and I've asked this before at least twice). Would the Archive be willing to consider upping the 48-hour window for uploads to, say, 96 hours??? Not that this would solve the problem I'm having right now. But, assuming I'm just encountering a glitch (as opposed to this being "normal"), 96 hours would give sincere uploaders much needed breathing space. Many of us (like me) have regular jobs and can't sit in front of their computers for double-digit hours to make sure their uploads complete. And many of us would probably like to turn their computers off when they're at work.

This post was modified by AlecWest on 2006-09-23 14:03:28

Reply to this post
Reply [edit]

Poster: MatCarpenter Date: Dec 11, 2006 7:49pm
Forum: forums Subject: Re: Is there a '24-hour rule' at Archive.org?

This from Dosdragon.com

[quote]
November 17th 2006 Comcast.net It has come to my attention that Comcast.net has been throttling users by intermittently shutting off a PC's access to the internet. This throttling is intentional, it does not slow the connection but actually HALT or shut off the connection for up to five minutes at a time. I believe they are doing this to prevent flooding, bittorent, down-loaders or VPN access. Though Comcast's terms of service prohibit the use of a VPN on a non-commercial account, their actions are hurting innocent customers. The intentional shut off a user's account can also effect Internet Telephones such as Vontage. These actions could disconnect an important call like 911. After studying the problem I found Comcast is storing MAC address's from personal PCs. This is a gross invasion of privacy and can land the cable company in court. I have provided software for users with a router that allows spoofing the MAC address and restores the connection. The software can be run manually or automatic to detect the loss of internet connection then reset your Network interface card with a new Spoofed MAC address restoring your connection within seconds. You may download the software here http://www.dosdragon.com/antithrottle.zip
[/quote]

http://www.dosdragon.com/antithrottle.zip

Reply to this post
Reply [edit]

Poster: Baire Date: Jun 1, 2007 11:06am
Forum: forums Subject: Re: Is there a '24-hour rule' at Archive.org?

Mat, your link is broken or comcast is not allowing me to visit! I'm looking for your program PLEASE HELP! If this is true this could be the begining of a class action?

Reply to this post
Reply [edit]

Poster: Administrator, Curator, or Stafftracey pooh Date: Feb 23, 2008 4:39pm
Forum: forums Subject: Re: Is there a '24-hour rule' at Archive.org?

you know, i can also say that it's clear comcast is absolutely doing something.

being IA webmaster, i can directly upload via ssh/scp or rsync to our servers files in the 100s+ MB range.

what i see is a very fast upload for about the first 10-20 seconds, and then it plummets. EVERY time. interestingly, if i want to make my upload go faster, when i watch the upload bitrate plummet, i can "pause" the upload, wait about 30+ seconds, then resume. and i see the exact same "FAST" for a bit, then plummets. so it's clearly comcast (my provider).

i suspect we are all paying for X b/sec uploads, but if we all actually used it, they'd run out of headroom. thus, they put in throttling.

--tracey


Reply to this post
Reply [edit]

Poster: Administrator, Curator, or StaffDiana Hamilton Date: Feb 23, 2008 7:09pm
Forum: forums Subject: Re: Is there a '24-hour rule' at Archive.org?

Latest:
http://www.news.com/Comcast-vs.-BitTorrent-to-be-focus-of-FCC-hearing/2100-1028_3-6231737.html

Comcast vs. ftp has not been directly in the news so much or in governamental sights, but it makes sense that they might throttle multiple protocols if they were messing with one.

This post was modified by Diana Hamilton on 2008-02-24 03:09:15